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RENDERING A PENCIL- SKETCH IMAGE 



TECHNICAL FIELD 

This invention relates to rendering a pencil-sketch image 
from three-dimensional (3D) data. 

BACKGROUND 

A pencil-sketch image approximates shading and depth by 
varying the placement and density of discrete line segments. 
Unlike traditional "smooth", or Gouraud, shading where 
transitions between light and dark regions of an image are 
gradual, pencil-sketching uses hard edge boundaries between 
regions. That is, transitions between regions are created by 
terminating line segments in the regions, not by blending one 
neighboring region into another region. 

DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a view of a Gouraud-shaded 3D model. 
Fig. 2 is a wireframe view of polygons in the 3D model. 
Fig. 3 is a view of one of the polygons. 
Fig. 4 is a view of light hitting the polygon. 
Fig. 5 is a flowchart of a process for generating a 
pencil-sketch image from the polygon. 
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Figs. 6 to 10 shows textures used for the pencil-sketch 
image . 

Fig. 11 shows a background for the pencil-sketch image. 

Fig. 12 is a view showing how a perspective of a 3D model 
is mapped onto a two-dimensional (2D) surface. 

Fig. 13 shows two pencil-sketch images rendered by the 
process of Fig. 5. 

Fig. 14 is a block diagram of a computer system on which 
the process of Fig. 5 may be implemented. 



M DESCRIPTION 

Uj 

SO Fig. 1 shows a 3D model 10. 3D model 10 is a Gouraud- 

J*j shaded model defined by 3D data. As shown in Fig. 2, the 3D 

;\. data defines interconnecting polygons 11, which comprise 3D 

h 15 model 10. Polygons 11 are triangles in this embodiment; 
f*\ however, other types of polygons may be used to construct the 

3D model. Groups of polygons are organized into meshes, each 
of which corresponds to an element of the 3D model. 

Referring to Fig. 3, the 3D data for a polygon 13 is 
20 comprised of coordinates for three vertices 15a, 15b and 15c 
positioned in Cartesian XYZ (or other) space. These vertices 
define a face 16 and edges 17a, 17b and 17c for the polygon. 
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A unit normal vector ("normal") 20a, 20b and 20c at each 
respective vertex 15a, 15b and 15c affects how the vertex is 
perceived relative to a predefined reference point (the 
"eyepoint") 23 (Fig. 4) in the "virtual world" that 3D model 
10 inhabits. Taking vertex 15b as an example in Fig. 4, 
normal 20b determines the amount of light that reaches vertex 
15b from a predefined light source 24 in the virtual world. 
The amount of light is determined using the dot product of 
unit normal 20b and a unit vector 25 from the light source. 
The dot product value defines the cosine of angle 18 between 
the light and the normal. The shading applied to each polygon 
face is determined based on this angle, as described below. 
Coordinates for the normals may be stored with the 3D data for 
each vertex. Alternatively, a normal may be computed "on-the- 
fly" during pencil-sketch image rendering. 

Fig. 5 shows a process 27 for rendering pencil-sketch 
images from a 3D model. There are two phases to process 27: 
a pre-processing phase 29 and a run-time phase 30. 

In pre-processing phase 29, process 27 obtains (51) a set 
of "pencil" markings. The pencil markings are bitmap images 
of line segments that may be scanned-in, read from a disk, 
retrieved from memory, or generated dynamically. The pencil 
markings may be straight, curved, or crooked. Also, the 
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pencil markings may be of varying thickness and length, 
depending upon the type of textures that they are used to 
construct . 

Process 27 constructs (52) a set of N (N>1) 2D textures 

5 by selecting pencil markings and arranging them uniformly to 
create various texture maps/tiles. The pencil markings are 
arranged at different densities and are parallel and/or 
perpendicular to one another to create different textures. 
Figs. 6 to 10 shows different types of textures that were 

10 constructed by arranging pencil markings. 

In Fig. 6, the pencil markings are arranged at a low 
density and only in the Cartesian X-coordinate direction. 
Figs. 7 and 8 show higher density versions of the texture 
shown in Fig. 6. In Fig. 9, the pencil markings are arranged 

15 in both the Cartesian X and Y directions (i.e., the pencil 

markings are cross-hatched) and at a relatively high density. 
Fig. 10 shows a higher-density version of the texture of Fig. 
9. More, less and/or different textures may be used with 
process 27. For example, a blank texture, which includes no 

20 pencil sketch markings, may be used. Since the textures are 
tiled, the textures may be constructed so that there is 
continuity between the end point of a line segment on one tile 
and the start point of a line segment on an adjacent tile. 
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Thus, when creating the line segments, it is preferable to 
ensure that the Co continuity property holds, where the Co 
continuity property is defined as having the tangent vectors 
of two curve segments be equal (in both direction and 
magnitude) at the segments 1 joint (or intersection) point. 

However, this does not always alleviate the appearance of 
tiling; accordingly, the line segments may be created to 
ensure that the Ci continuity property holds, in which the 
first derivatives (slopes) of the segments at the start and 
end points of adjacent tiles are roughly equal. This can be 
difficult to achieve, but can be simulated by randomly 
selecting the starting point for a line segment and wrapping 
the line segment around the texture at the end of the tile. 

Pre-processing phase 29 set forth above may be performed 
at any time prior to run-time phase 30. It is noted that a 
single pre-processing phase may be used to store textures for 
several different run-time phases. 

In run-time phase 30, process 27 selects (53) a 
background onto which a pencil-sketch image is to be rendered. 
The background may be selected from a set of backgrounds 
stored in memory or it may be obtained from another source, 
such as a disk or a scanned image. The background is an 
orthographic projection of a relatively large quadrilateral 
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texture mapped with a paper (or other) texture. Fig. 11 shows 
an example of a background; however, other backgrounds, or 
even no background, may be used with process 27. 

When rendering a pencil-sketch image from 3D polygon 
data, process 27 determines (54) which pencil-sketch texture 
to use for the polygon. Process 27 does this based on the way 
that the polygon is illuminated, i.e., based on the light that 
hits the polygon. To determine how light hits a polygon, 
process 27 obtains (55) a texture value using the vertex 
normals (see Fig. 3). For polygon 13 (Fig. 4), process 27 
calculates the vector dot product of unit normal vector 20b 
(N) and unit light vector 25 (L) . 

Since N and L are both unit vectors the product of N ■ L is 
the cosine of the angle 18 formed between the two vectors. If 
the angle between N and L is small, then the diffuse component 
of smooth shading is high and N-L will have a value close to 
one. On the other hand, if the angle is large, then the 
amount of diffuse component in smooth shading is low and N-L 
has a value close to zero. 

Process 27 takes the maximum of the resulting dot product 
(N-L) and zero, i.e., Max (N-L, 0) and defines that value as the 
texture value for the vertex, in this case vertex 20b of 
polygon 13. The maximum is taken to discount polygons that 
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are in the back of the 3D model relative to the light source 
and, thus, produce a negative N-L value. 

For each vertex 20a, 20b and 20c of polygon 13, process 
27 obtains (55) a texture value. Process 27 classifies (56) 
the polygon based on the obtained texture values. Process 27 
uses the texture values to associate each vertex of polygon 13 
with one of M (M>1) bins in memory, each of which corresponds 
to a predetermined range of values. For example, a system 
might include three bins having intervals of [0,a], (a,b] and 
(b,l], where "a" and "b" are adjustable values with a<b, 0<a 
and b^l, and where square brackets indicate exclusion and 
parenthetic brackets indicate exclusion, e.g., "a" is included 
in the range [0,a] but excluded from the range (a,b]. So, in 
this example, if a texture value of vertex 20b is "a", vertex 
20b will be associated with bin [0 f a]. Different numbers 
and/or ranges of bins may be used in process 27. 

Process 27 associates (57) one of the N pencil sketch 
textures from Figs. 6 to 10 with polygon 13 based on the 
classifications of the polygon's vertices. Process 27 builds 
n (n>l) face lists in memory, each of which corresponds to one 
of the N textures ( "N" here is not necessarily equal to "n"), 
and assigns polygon 13 to one of those face lists based on the 
bins into which the polygon's vertices fall. For polygon 13, 
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if each vertex 20a, 20b and 20c falls in the same bin, the 
polygon is appended to a face list that correlates to the bin. 
If different vertices of polygon 13 fall into different bins, 
then the polygon is appended to the most appropriate face 
list. For example, if two vertices belong to the same bin, 
but one other vertex does not, the polygon may be appended to 
the face list for that bin despite the other vertex. 

Once process 27 determines (54) the texture for polygon 
13, process 27 projects (58) polygon 13 onto a 2D surface. 
Referring to the example shown in Fig. 12, this is done by 
determining the XY coordinates on 2D surface 30 (e.g., a 
computer monitor) of a polygon 31 on 3D model 32. Process 27 
projects the coordinates of the polygon onto 2D surface 30, 
resulting in a 2D representation of the polygon. 

Referring back to Fig. 5, process 27 maps (59) the 
appropriate texture onto the 2D representation of polygon 13. 
As noted, the texture of polygon 13 is determined based on the 
face list to which polygon 13 is appended. Process 27 is 
repeated for each polygon in a 3D model, resulting in a 
pencil-sketch image of the 3D model. Examples of pencil- 
sketch images generated by process 27 are shown in Fig. 13. 

Process 27 may be used to create animation eels for 
cartooning. For example, a 3D model, such as model 10, may be 
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generated, and then positioned in a desired manner. Process 
27 may be executed on the model to produce a pencil-sketch 2D 
image for that position . Then, the 3D model 10 can be re- 
positioned (e.g., rotated), and process 27 executed on the re- 
positioned model to produce a pencil-sketch 2D image for a 
different perspective of the model. This process may be 
repeated to produce pencil-sketch 2D images for any number of 
model positions. Thus, process can generate animation eels 
automatically, meaning without the use of hand-drawn sketches. 

Process 27 runs in real-time, which facilitates the 
animation process. That is, in conventional hand-drawn 
animation, artists cannot interactively change the 
appearance/view of a character without re-drawing the 
character manually. Process 27 permits this because it 
renders frames of animation (i.e., 2D images) dynamically and 
automatically for a given viewpoint in real-time. In this 
regard, the viewpoint is not the only aspect of a frame that 
can be dynamically manipulated using process 27. Light moving 
relative to a character and model changes the locations of 
shadows on those objects, just as in a conventional 3D 
Gouraud-shaded scene . 

Process 27 can be used for interactive technical 
illustrations and real-time video game play. For example, a 
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pencil-sketch game may be constructed in which a user 
navigates throughout a virtual world that appears in 2D, e.g., 
a world that simulates a newspaper comic. So-called "How-To" 
manuals, particularly the online variety, often make use of 
pencil-sketch drawings to illustrate aspects of a model. 
Process 27 may be used to allow a reader to examine the model 
from different angles/perspectives. 

Fig. 14 shows a computer 35 for rendering pencil-sketch 
images using process 27. Computer 35 includes a processor 36, 
a memory 37, a storage medium 39 (e.g., a hard disk), and a 3D 
graphics accelerator card 40 for repositioning a 3D model and 
manipulating 3D data (see view 41) . Storage medium 39 stores 
3D data 42 which defines a 3D model, and computer instructions 
44 which are executed by processor 36 out of memory 37 to 
render pencil-sketch images using process 27 and 3D data 42. 
Memory 37 also stores the face lists and bins noted above. 

Process 27 is not limited to use with the hardware and 
software of Fig. 14; it may find applicability in any 
computing or processing environment and with any type of 
machine that is capable of running a computer program. 
Process 27 may be implemented in hardware, software, or a 
combination of the two. Process 27 may be implemented in 
computer programs executing on programmable computers that 
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each include a processor, a storage medium readable by the 
processor (including volatile and non-volatile memory and/or 
storage elements), at least one input device, and one or more 
output devices. Program code may be applied to data entered 
using an input device to perform process 27 and to generate 
output information . 

Each such program may be implemented in a high level 
procedural or object-oriented programming language to 
communicate with a computer system. However, the programs can 
be implemented in assembly or machine language. The language 
may be a compiled or an interpreted language. 

Each computer program may be stored on a storage medium 
or device (e.g., CD-ROM, hard disk, or magnetic diskette) that 
is readable by a general or special purpose programmable 
computer for configuring and operating the computer when the 
storage medium or device is read by the computer to perform 
process 27. Process 27 may also be implemented as a computer- 
readable storage medium, configured with a computer program, 
where, upon execution, instructions in the computer program 
cause the computer to operate in accordance with process 27. 

Other embodiments not described herein are also within 
the scope of the following claims. 

What is claimed is: 
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1. A method of rendering a pencil-sketch image from 
three-dimensional data, comprising: 

determining a pencil-sketch texture for a polygon defined 

by the three-dimensional data; 

projecting the polygon onto a two-dimensional surface; and 
mapping the pencil-sketch texture onto the polygon to 

render the pencil-sketch image. 

2. The method of claim 1, further comprising: 
obtaining a set of pencil-sketch markings; and 
constructing the pencil-sketch texture using the pencil- 
sketch markings. 

3. The method of claim 2, wherein constructing comprises 
arranging the pencil sketch markings at a predetermined 
density such that the pencil sketch markings are at least one 
of parallel and perpendicular to one another. 

4. The method of claim 1, further comprising: 
selecting a background onto which the pencil sketch image 

is rendered. 
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5. The method of claim 1, wherein determining the 
pencil-sketch texture comprises: 

obtaining a value based on a normal vector to the polygon 
and a light vector between the polygon and a light source; 
classifying the polygon based on the value; and 
associating the pencil-sketch texture with the polygon 
based on the classification of the polygon. 

6. The method of claim 5, wherein the normal vector is a 
vertex normal of the polygon. 

7. The method of claim 5, wherein the value is obtained 
by calculating the vector dot product of the normal vector and 
the light vector. 

8. The method of claim 5, wherein the polygon is 
classified by associating the polygon with one of M (M>1) 
bins, each of the M bins corresponding to a predetermined 
range of values. 

9. The method of claim 8, wherein the pencil-sketch 
texture is associated with the polygon based on which of the N 
bins into which the polygon falls. 
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10. The method of claim 1, wherein the three-dimensional 
data defines a three-dimensional model; and 

the method further comprises: 

re-positioning the three-dimensional model relative to 
the two-dimensional surface; and 

repeating determining, projecting and mapping for a 
second polygon on the three-dimensional model to render a 
second pencil-sketch image. 

11. An article comprising: 

a readable medium that stores executable instructions to 
render a pencil-sketch image from three-dimensional data, the 
instructions for causing a machine to: 

determine a pencil-sketch texture for a polygon defined by 
the three-dimensional data; 

project the polygon onto a two-dimensional surface; and 

map the pencil-sketch texture onto the polygon to render 
the pencil-sketch image. 

12. The article of claim 11, further comprising 
instructions that cause the machine to: 

obtain a set of pencil-sketch markings; and 
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construct the pencil-sketch texture using the pencil- 
sketch markings. 

13. The article of claim 12, wherein constructing 
comprises arranging the pencil sketch markings at a 
predetermined density such that the pencil sketch markings are 
at least one of parallel and perpendicular to one another. 

14. The article of claim 11, further comprising 
instructions that cause the machine to: 

select a background onto which the pencil sketch image is 
rendered. 

15. The article of claim 11, wherein determining the 
pencil-sketch texture comprises: 

obtaining a value based on a normal vector to the polygon 
and a light vector between the polygon and a light source; 
classifying the polygon based on the value; and 
associating the pencil-sketch texture with the polygon 
based on the classification of the polygon. 

16. The article of claim 15, wherein the normal vector 
is a vertex normal of the polygon. 
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17. The article of claim 15, wherein the value is 
obtained by calculating the vector dot product of the normal 
vector and the light vector. 

18. The article of claim 15, wherein the polygon is 
classified by associating the polygon with one of M (M>1) 
bins, each of the M bins corresponding to a predetermined 
range of values. 

19. The article of claim 18, wherein the pencil-sketch 
texture is associated with the polygon based on which of the N 
bins into which the polygon falls. 

20. The article of claim 11, wherein the three- 
dimensional data defines a three-dimensional model; and 

the article further comprises instructions that cause the 
machine to: 

re-position the three-dimensional model relative to the 
two-dimensional surface; and 

repeat determining, projecting and mapping for a second 
polygon on the three-dimensional model to render a second 
pencil-sketch image . 
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21. An apparatus for rendering a pencil-sketch image 
from three-dimensional data, comprising: 

a memory that stores executable instructions; and 
a processor that executes the instructions to: 

determine a pencil-sketch texture for a polygon 
defined by the three-dimensional data; 

project the polygon onto a two-dimensional surface; 

and 

map the pencil-sketch texture onto the polygon to 
render the pencil-sketch image. 

22. The apparatus of claim 21, wherein the processor 
executes instructions to: 

obtain a set of pencil-sketch markings; and 
construct the pencil-sketch texture using the pencil- 
sketch markings. 

23. The apparatus of claim 22, wherein constructing 
comprises arranging the pencil sketch markings at a 
predetermined density such that the pencil sketch markings ar 
at least one of parallel and perpendicular to one another. 
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24. The apparatus of claim 21, wherein the processor 
executes instructions to: 

select a background onto which the pencil sketch image is 
rendered. 

25. The apparatus of claim 21, wherein determining the 
pencil-sketch texture comprises: 

obtaining a value based on a normal vector to the polygon 
and a light vector between the polygon and a light source; 
classifying the polygon based on the value; and 
associating the pencil-sketch texture with the polygon 
based on the classification of the polygon. 

26. The apparatus of claim 25, wherein the normal vector 
is a vertex normal of the polygon. 

27. The apparatus of claim 25, wherein the value is 
obtained by calculating the vector dot product of the normal 
vector and the light vector. 

28. The apparatus of claim 25, wherein the polygon is 
classified by associating the polygon with one of M (M>1) 
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bins, each of the M bins corresponding to a predetermined 
range of values. 

29. The apparatus of claim 28, wherein the pencil-sketch 
texture is associated with the polygon based on which of the N 
bins into which the polygon falls. 

30. The apparatus of claim 21, wherein the three- 
dimensional data defines a three-dimensional model; and 

the processor executes instructions to: 

re-position the three-dimensional model relative to the 
two-dimensional surface; and 

repeat determining, projecting and mapping for a second 
polygon on the three-dimensional model to render a second 
pencil-sketch image. 
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RENDERING A PENCIL-SKETCH IMAGE 
ABSTRACT 

A pencil-sketch image is rendered from three-dimensional 
(3D) data by determining a pencil-sketch texture for a polygon 
defined by the 3D data, projecting the polygon onto a two- 
dimensional surface, and mapping the pencil-sketch texture 
onto the polygon to render the pencil-sketch image. The 
pencil-sketch texture is determined by obtaining a value based 
on a normal vector to the polygon and a light vector between 
the polygon and a light source, classifying the polygon based 
on the value, and associating the pencil-sketch texture with 
the polygon based on the classification of the polygon. 

20135951.doc 
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1001 of Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the 
application or any patents issued thejjpon. 

Full Name of Inventor: ADAM T. 




Inventor's Signature: Date 

Residence Address: BtSaverton, OR 97006 " " " ' 

Citizenship: U T S. 

Post Office Address: 16691 NW Avondale Drive 
Beaverton, OR 97006 
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Full Name of Inventor: MARC S. BLACKSTEIN 

Inventor's Signature: m Date; 

Residence Address: Hillsboro, OR 97 124 
Citizenship: U,Sr 

Post Office Address; 3063 NW Overlook Dr., Apt. 635 
Hillsboro,OR 97124 



Pull Name of Inventor: CARL S. MARSHALL 




Inventor' s Signature; ' &*>d!$\ Date: //- 3~<a> 

Residence Address; Portland, OR 97229 
Citizenship: U,S. 

Post Office Address: 1 545 1 NW Andalusian Way 
Portland, OR 97229 



Full Name of Inventor: 

Inventor's Signature: 
Residence Address; 
Citizenship: 
Post Office Address; 




Date; 



Hillsboio, OR 971 
U.S. 

545 NE 26th Court 
Hillsboro, OR 97124 
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Attorney's Docket No-: 10559-296001 
Client's Ref, No.; F9307 

COMBINED DECLARATION AND POWER OF ATTORNEY 

As a below named inventor, I hereby declare that 

My residency post office address and citizenship are as seated below next to my name. 

T believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and 
joint inventor (if plural names are listed below) of the subj ect matter which is claimed and for which a patent is 
sought on the invention entitled RENDERING A PENCIL-SKETCH IMAGE, the specification of which: 

[x] is attached hereto. 

[] was filed on _ as Application Serial No. _ and was amended on 

[] was described and claimed m PCT International Application No, filed on 

. and as amended under PCT Article 19 on , 

I hereby state that I have reviewed and understand the contents of the above-Identified Specification, 
including the claims, as amended by any amendment referred to above, 

I acknowledge the duty to disclose all information T know to be material to patentability in accordance with 
Title 37, Code of Federal Regulations, § 3 .56. 

I hereby appoint the following attorneys and/or agents to prosecute this application and to transact all 
business in the Patent and Trademark Office connected therewith: 

Scott C, Harris, Reg. No. 32,030 David L. Feigenbaum, Reg. No. 30,378 

Halph A. Mittelberger, Reg, No. 33, 195 Hans R. Troesch, Reg, No. 36,950 

John R. WetherelL Jr., Reg, No. 3 1 ,678 Bing Ai, Reg. No. 43,3 12 

Paul A. Pysher, Reg. No. 40,780 Samuel L. Lee, Reg, No. 42,791 

Frederick H. Rabin, Reg, No. 24 7 488 John C. Phillips, Reg. No. 35,322 

Joseph R. Baker, Reg. No. 40,900 Richard J, Anderson, Reg. No. 36,732 

Samuel Borodach, Reg. No. 38,358 William J, Egan, JIT ? Reg. No. 28,41 1 

Address all telephone calls to Scott C Harris at telephone number (85S)/678-5070. 

Address all correspondence to Scott C. Harris at: FISH & RICHARDSON P.C., 4350 La Jolla Village 
Drive, Suite 500, San Diego, CA 92122, 

1 hereby declare that all statements made herein of my own knowledge are true and that all statements made 
on information and belief are believed to be true; and further that these statements were made with the knowledge 
lhat willful false statements and the like so made are punishable by fine or imprisonment, or both, under Section 
1001 of Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the 
application or any patents issued thereon. 

Full Name of Inventor ADAM T. LAKE 

Inventor's Signature: Date: 

Residence Address: Beaverton 7 OR 97006 
Citizenship: U.S. 
Post Office Address: 1 669 1 NW Avondale Drive 
Beaverton- OR 97006 
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Full Name of Inventor: 

Inventor's Signature: 
Residence Address; 
Citizenship: 
Post Office Address: 



MARC S. BLACKSTETN 




Hillseorb, OR 97124 
U.S. 

3063 NW Overlook Dr„ Apt 635 
Hillsboro, OR 97124 



Full Name of Inventor: CARL S. MARSHALL 



Inventor's Signature: 
Residence Address: 
Citizenship: 
Post Office Address; 



Portland, OR 97229 
U.S. 

15451 NW Andalusian Way 
Portland* OR 97229 



Date: 



Date; 



Full Name of Inventor: DANIEL JOHNSTON 



Inventor's Signature: 
Residence Address: 
Citizenship; 
Post Office Address: 



Hfflsboro,OR97124 
U.S- 

545 NE 26th Court 
Hillsboro, OR 97 124 



Date: 



20l4I900.doc 



